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I. INTRODUCTION 

Topological insulators in three dimensions (3D) are 
band insulators which have a bulk insulating gap and 
gapless surface states with odd number of Dirac cones 
protected by time-reversal symmetry (TRS)ii"— A fam- 
ily of 3D topological insulators (TI) with a large bulk 
gap and a single Dirac cone on the surface includes the 
compounds Bi2Se3, Bi2Te3 and Sb2Te3, which have been 
theoretically predicted and experimentally observedi^^— 
The surface state of these materials can be described by 
the effective Dirac Hamiltonian Hq = Tivpz ■ {cr x k) 
(with k = (kx, ky) the momentum) when the Fermi level 
is close to the Dirac point, which behaves like a mass- 
less relativistic Dirac fermion with the spin locked to its 
momentunifS However, compared to the familiar Dirac 
fermions in particle physics, those emergent quasiparti- 
cles exhibit richer behaviors. In Bi2Te3, an unconven- 
tional hexagonal warping effect appears due to the crys- 
tal symmetry)^ which means the constant energy contour 
(CEC) of the surface band evolves from a convex circle 
to a concave hexagon as the energy moves away from the 
Dirac point. Although the topological property of the 
surface states is not affected, such kind of deformation of 
the CEC does affect the behavior of the surface states in 
the presence of impurities. 

Quasiparticle interference (QPI) caused by impurity 
scattering on the surface of 3DTIs is an effective way 
to reveal the topological nature of the surface states. 
The interference between incoming and outgoing waves 
at momenta k^ and k^ leads to an amplitude modula- 
tion of the local density of state (LDOS) at wave vec- 



tor q 



ki, known as the Friedel oscillation 
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Nowadays, such modulation can be studied by a pow- 
erful surface probe, the scanning tunneling microscopy 
(STM), which directly measures the LDOS. The infor- 
mation in momentum space is obtained through Fourier 
transform scanning tunneling spectroscopy (FT-STS). 
Several STM measurements^^— have been performed 
on the surface of 3DTIs in the presence of nonmagnetic 



point and edge impurities, and the following features are 
shared in common, (i) The topological suppression of 
backward scattering from nonmagnetic point and edge 
impurities is confirmed by the observation of strongly 
damped oscillations in LDOS, together with the invisibil- 
ity of the corresponding scattering wave vector q in FT- 
STS. (ii) Anomalous oscillations are reported in Bi2Te3 
for both point and edge impurities when the CEC be- 
comes concave. These experimental facts have been in- 
terpreted theoretically by several groups J^"— For short- 
range point and edge impurities, the Friedel oscillation 
in an ordinary two-dimensional electron gas (2DEG) has 
the power law of and R^^/^ respectivelyi^ In com- 
parison, the Friedel oscillation in a helical liquid with 
a convex CEC is dominated by the scattering between 
time-reversed points (TRP) and is thus suppressed to 
R^^ and R~^/'^ for point and edge impurities separately. 
This result is the crucial reason of the invisibility of the 
scattering wave vector q in FT-STS, and is the direct con- 
sequence of the suppression of backscattering protected 
by TRS in helical liquid. When the CEC becomes con- 
cave, scattering between wave vectors, which are not con- 
nected by TRP, can have a significant contribution and 
leads to a slower decay of the Friedel oscillation<^ii^ 

Motivated by these results, in this work we develop a 
general theory of the QPI for a CEC of generic shape us- 
ing the stationary phase approximation approachi^ This 
approach has been applied successfully to the Ruderman- 
Kittel-Kasuya-Yosida interaction in 3D systems with 
nonspherical Fermi surfaces <^ In the stationary phase 
approximation, the long distance behavior of the Friedel 
oscillations is dominated by the so-called "stationary 
points" on the CEC. Using this approach, a complete 
result of the power-expansion series of the LDOS and 
spin LDOS is obtained for both point- and edge-shaped 
nonmagnetic and magnetic impurities, which we model 
by 5-function potentials. The spin LDOS is the local 
spin density at a given energy, which can be measured 
by a STM experiment with a magnetic tip. Our results 
depend only on the TRS and the local geometry around 
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FIG. 1: (Color online) Illustration of charge and spin interfer- 
ence patterns between two counter-propagating helical waves. 
The gray block is a 3DTI with a magnetic edge impurity 
(green stripe) lying along the a;-axis on the surface. An in- 
cident helical wave along y-direction with spin polarized in 
the x-direction (blue line) is backscattered by the magnetic 
edge and the spin is flipped (red line). The interference of 
the two orthogonal helical waves leads to a constant LDOS 
in the charge channel, but a spiral LDOS in the spin channel 
(purple arrows) in the yz-plaxie. 

the stationary points on the CEC, which explain not only 
the usual and power laws in 2DEG but also 

the and R~^/'^ oscillations in the helical liquid. With 
a generic shape of CEC, a different power law can be ob- 
tained due to the presence of additional stationary points 
besides the TRP, which can be used to predict the result 
of STM and spin-resolved STM experiments on the sur- 
face of other TI materials with more complicated surface 
states. An important consequence of our result is that 
an ordinary STM measurement cannot distinguish mag- 
netic and nonmagnetic impurities, although the former 
can induce backscattering while the latter cannot. To 
distinguish the effect of magnetic and nonmagnetic im- 
purities and observe backscattering induced by magnetic 
impurities, it is necessary to use a magnetic tip to mea- 
sure the spin LDOS. 

The rest of this paper is organized as follows. In Sect. 
im we introduce an intuitive picture of the interference 
between helical waves scattered by magnetic impurities. 
In Sect. Illli we present the general analytic formulation 
of LDOS for point and edge impurities respectively by 
focusing firstly on those CEC where the stationary points 
are extremal points. We then generalize our results to 
the more generic CEC where the stationary points are 
saddle points, with the first nonzero expansion coefficient 
occurring at a higher power. Conclusion and discussion 
are given in Sect. IIVI 

II. STANDING WAVE OF THE SPIN 
INTERFERENCE BETWEEN TWO HELICAL 
WAVES 

With the presence of TRS, the backscattering by non- 
magnetic impurities is known to be forbidden on the sur- 
face of 3DTIs, due to the tt Berry's phase associated with 
the full rotation of electron spin.— In experiments, this 
manifests in the invisibility of the scattering wave vector 



2kF in FT-STSJ^ It would then be interesting to ask 
how the surface states respond differently to magnetic 
impurities, and what are their characteristic signatures in 
STM measurements. With magnetic impurities, naively 
one would expect to see a nontrivial interference pattern 
since backscattering is allowed due to the breaking of 
TRS. However, it turns out that the Friedel oscillation 
in the charge LDOS, which is measured in an ordinary 
STM experiment with a nonmagnetic tip, is still sup- 
pressed in the same way as nonmagnetic impurities. The 
broken TRS would only manifest itself in the spin LDOS 
measured by a spin-polarized STM tipi^ 

To understand this result, we first present a simple pic- 
ture of the interference between two counter-propagating 
helical waves on the surface of a 3DTI, and then give 
a complete theoretical survey in the next section. Con- 
sider a magnetic edge impurity placed along the x-axis on 
the surface. For the effective Hamiltonian Hq = fivpz ■ 
(<T X k) , the electron state propagating along y direction 
perpendicular to the impurity line has spin polarized to 
x-direction, with the wavefunction V'l = l)"""- 
Here the superscript "T" indicates the transpose. This 
wave is then backscattered by the magnetic edge and 
counter-propagates in — j/-direction. For the same en- 
ergy, the state with opposite k must have opposite spin, 
with the wavefunction ?/'2 = 1)'^. This sit- 

uation is illustrated in FigHJ A simple calculation shows 
that the interference of the two counter-propagating he- 
lical waves, Vly) = "Tf'-^'i^^-' ^ ^^{y)), leads to a con- 
stant charge LDOS on the surface (p)^ = \'ip^'ip{y)\ = 1 
since "01 and V'2 have orthogonal spin. However the in- 
terference leads to a spiral spin LDOS in yz-plane as 
(s)^ = ■0^8-0 = [0, — J sin{2kpy), — J cos{2kpy)], where 
s = |cr is the electron spin operator. Therefore a STM 
experiment with a nonmagnetic tip will observe no in- 
terference pattern while one with a magnetic tip will ob- 
serve the oscillation of the spin density of states. Such 
a contrast between charge and spin density of states is a 
unique signature of the helical liquid, which is a direct 
demonstration of the locking between spin and momen- 
tum. 

To observe such a spin interference pattern, a more 
convenient setup is a closed "magnetic wall" as shown in 
Fig 12] Consider a magnetic layer deposited everywhere 
on the 3DTI surface except a hole in the middle with 
the disk shape. The magnetic layer can open a gap on 
the surface state, such that the low energy surface states 
are trapped in the hole region and form standing wave. 
Similar to the straight line magnetic impurity discussed 
above, the standing wave trapped by the magnetic bar- 
rier can be obtained by setting the boundary condition 
of fixed spin at the boundary of the hole. For large R 
{R ^ 1/kp) the spin density of the standing wave has 
the behavior of (sr) ~ SiifeR), (s,) ^ S2S(2^^ with 
R and z standing for longitudinal and perpendicular di- 
rections in a spherical coordinate. A unique property 
of the helical surface states is the spin-charge locking^. 
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FIG. 2: (Color online) Standing wave of spin interference be- 
tween two helical waves inside a closed "magnetic wall" on 
top of a 3DTI surface. The magnetic wall is surrounded by a 
magnetic layer deposited on top of the 3DTI surface, which 
opens a gap in the helical surface states and plays the role 
of a barrier. The out-of-plane spin LDOS is exhibited by the 
colored rings, and the in-plane spin LDOS is indicated by the 
dark arrows. 



For the effective Hamiltonian Hq = hvpz ■ {cr x h), the 
electric current operator in the long wavelength limit is 
j = Vk-ffo = ^vpz X <T. Therefore there is a loop charge 
current = —2evp{sii) along the azimuthal direction 
associated with the spin density. 



III. GENERAL FORMULATION OF 
STATIONARY PHASE APPROXIMATION 
APPROACH TO QPI ON THE SURFACE OF 
3DTI 

In this section, we obtain the general long-distance fea- 
tures of charge and spin LDOS on the surface of a generic 
3DTI induced by nonmagnetic and magnetic impurities 
using the stationary phase approximation methodi^ We 
study both point-like and edge-like impurities. We shall 
focus first on the behavior of a special kind of CEC where 
the stationary points arc extremal points, and then gen- 
eralize our results to generic CEC with higher order nest- 
ing points. 



A. Point impurity 

We start by considering a point defect on the surface 
of a 3DTL The Hamiltonian with a single impurity is 



H 



d2rV't(r) [ho{k) + Va^Sir)] il;{r), 



(1) 



where cr^ = 1 for /i = and cr" = a-^-y-- is the Pauli 
matrices for a = 1,2,3. k = — iV is the momentum 
operator. For such a potential the LDOS can be ex- 
pressed exactly. Using the <t^ matrices, the charge and 
spin LDOS are combined to the form 



1 



Im{tr[G''(w,R, Hy]}, 



(2) 



with G^{uj, R, R') being the retarded Green's function in 
real space. Let p,^q{uj) be the LDOS of the unperturbed 



system with V = 0, the deviation of the LDOS from the 
background value Pi^q(uj) is then given by 



ilm/^e^(-''')-x 



tr[GS(c.,k)T^(c.,k,k')GS(c.,k')an (3) 

Here GQ(w,k) is the free retarded Green's function gov- 
erning the CEC under consideration. For the topological 
surface states G;i(w,k) = {lo + iS - /io(k))"i. The T- 
matrix T^{lj, k, k') is defined by 



(4) 



which is momentum independent when the impurity has 
a (5-function potential, and we have denoted the real space 
Green's function Go(w) = / f^J^-i Gq{uj, k). As is required 
by the TRS, Gq{uj) is always proportional to the identity 
matrix. 

We first note that the spin LDOS induced by a non- 
magnetic impurity vanishes uniformly, i.e., Spoa = 
for a = 1,2,3. This is a direct consequence of TRS, 
because under timc-rcversal transformation = icr^, 
we have O-V^O = -cr°^ and O-iG?; .0 = GlC 



k'T'Gn 



;.cr°], where we have abbreviated 
Gg J.. By interchanging k and — k' in the in- 



then the trace in Eq.© satisfies tj:[Gl ,.T° G^ a"] = 
-tr[GS,_ 
GS(w,k) 

tegral in Eq.(l3]), one is led to the result (5poa(w,R) = 0. 
To obtain other components of the T-matrix, we expand 
the T-matrix into a spin-dependent and spin-independent 
parts as 



rpa 

a 



V 



l-y2Gr2(^)' 
V 

T^yGsM 



1 - V^-G^„'^{u) 



(5) 



where the fact that Gq{uj) is proportional to identity has 
been used, and no summation over repeated indices is 
implied throughout the paper. Similar to the argument in 
5poa case, we see that the contribution of to the charge 
LDOS of a magnetic impurity 6pao vanishes. Hence we 
have Spao/Spoo = Tg /T°. Therefore, in the following, we 
shall focus only on Spoo and Spab- 

To proceed, the measured LDOS in Eq.(l3|) is then 
rewritten in the diagonal basis of the topological sur- 
face bands. Define the unitary matrices L^k such that 
?7^/io(k)J7k diagonalizes /io(k), Eq.® becomes 

{ulT^Uk' ) (Ul Gl„ ) {Vl a^Uu^ (6) 
TT 7 (27r)4 
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FIG. 3: (Color online) Schematic picture of CEC and station- 
ary points for point and edge impurities, (a) Convex CEC 
where there is only one pair of stationary points connected 
by the red arrow along any given direction for both point and 
line impurities, (b) Concave CEC for point impurity where 
there are multiple pairs of stationary points. Nonstationary 
points are shown for example as blue arrows, (c) Concave 
CEC for edge impurities (brown line) where the slopes (green 
dashed lines) at the pair of stationary points are the same. 



The stationary points defined above include i) extremal 
points such as the pairs connected by qi in FigslSja) 
and (b), where the second derivative d'^ky/dk^ is nonva- 
nished; ii) the turning points such as the pair connected 
by q'2 in FiglSl^b), where the second derivative also van- 
ishes. In the following, we first focus only on the ex- 
tremal points, and leave the more general discussions to 

scct inrc] 

Having identified the pairs of stationary points on the 
CEC in direction R, the loop integrals in Eq.(|n]) at large 
distances are then approximated by the summation of 
integrals in the neighborhood of all the stationary-point 
pairs, which is the essence of the method of the sta- 
tionary phase approximation. To start with, we first 
change the integral variables as cPk = dedkxlTi\vyi\, 



where 



ds{'k)/hdkyi, and then expand the CEC at 



the extremal points as ky = kyi — (kx — k^i) /2pxi, where 
Pxi = —[d^kyiie^kx) / d'^kxi\~^ is the principle radii of 
curvature of the CEC at the extremal points, which is 
positive for maxima while negative for minima. Under 
this approximation, Eq.(|ni) becomes 



E 



{lo + i5- e„)(a; + i5 - e'^) ' 



where e„(m)(k) are the energy eigenvalues of the bands 
|n(m)k), and we have defined E^,„(k, k') = (nk|cr^|TOk'), 
as well as 



7C(k,k') 



rpay^a 
a nm 



/i = = 0, 



(8) 



Following the standard process of density of states 
calculations;^ the integrations over k and k' are then 
converted into coordinates dk = {dk±^,dk^) as 



1 

— Ito 

TT 



E 



dksdk'e'^^-^'y^ 



1. 



(7) Sp,.{co,R)c, J (i:y 



de' 1 



uj + i5 - e„ fi\vyi\ 



-ik'-R 



(27^)2c. + ^<5-e'^ n\v'A 



dx e ^f^i / dx e 
} J —00 



(11) 



k'xj , and all 



(27r)4 J |V^£„Vle^, 

7mn(k, k')I]5:'„„(k, k') 



{uj + iS- e„)(w + i5 - ej^) ' 



where k± and k^ are components of k normal and tan- 
gential to the CEC, respectively. 

To evaluate the loop integrals along the CEC, it is es- 
sential to introduce the stationary phase approximation. 
For example, consider the LDOS at a point R = Ry 
(here and hereafter we shall always take the y-dircction 
for example), the phase factor g^i^-'^'') '^ Q<ky-k'^)R^ 
Locally, one can write ky = ky{e, kx) as a function of en- 
ergy £ and kx- For large distance R from the impurity, 
the phase factors and e"'*^"^^ ■'^x)^ vary rapidly 

with respect to kx and k'^ for almost every point on the 
CEC, so that most of the integrations cancel out exactly 
except for the stationary points, k^^ which satisfy the 
condition 



where we have denoted x = kx~kxi, x' ~ k 
the quantities at the extremal points (ij) still depend on 
the energies e and e'. Now the matrix element I]^„(k, k') 
at the extremal points is in general some nonzero con- 
stant C^j^^(e,e'), except that it vanishes when p ~ 
and the pair of stationary points are time-reversal part- 
ners \nki) = QK\mk'.; 
jugation operator 
stationary points connected by qi's in FigslSja) and (b) 
for convex and concave CEC respectively. To obtain the 
generic behavior of the interference pattern, the matrix 
element is expanded in the distance x,x' to the stationary 
points as SJJ„(x, x') = C^.^^^ + ax + a'x' + o{x) + o(x'), 
where C^j = for /i = at TRP, and a nonvanishing 
but energy dependent constant otherwise. Inserting the 
series into Eq. pTj) . one can integrate first over x and x' 

dxe"^"^ 



.J I . Here K is the complex con- 
Examples are shown as the pairs of 



by using the relations 



and /^dra^e^^^' = 0F/(2|C|3/2)e-'i''g'i(C7), and then 
integrate over the energies using the residue theorem by 
summation over the integrand at the poles e — e' = 
Lo + iS. Finally by taking the limit w = e^, (5 — > 0"*" 
we get 



dkyie,kx) dk'y{e',k'x) 



dkx 



dk' 



0. 



(10) 



(5p^^(a;,R) 
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2TT'^h^R 



i-EE^' 



(kyi-k'.)R 



\Vy,v' 
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FIG. 4: (Color online) Fourier transformation of the LDOS 
with R^^ and R^^ power laws. 



X Ipii'Pi-'t'j) 'S^ Tf^C^ C^' -I- — 

s 

(12) 

where = —jsgn^p^i). This is the long wavelength 
behavior of LDOS induced by a point impurity. In 
the above result, we have s — and = = 
V/(\ — VGq{u))) for the charge LDOS of a nonmagnetic 
impurity (5poo- While for the spin LDOS of a magnetic 
impurity 8pab, the summation is over s = a, 0, where 
and Tg are respectively the spin-dependent and spin- 
independent coefficients in the T-matrix expansion intro- 
duced above. 

There are several comments regarding this result. 
Firstly, for a pair of non-TRS stationary points like c\2 
in Fig|3|b), the leading power is given by the first term 
in Eq.dn]), which is of i?"^ While for a pair of TRS 
stationary points as qi in FigsIS] (a) and (b), the first 
nonvanishing contribution to the power law is dominated 
by the second term in Eq. ([T2|) as R^^ for nonmagnetic 
impurity, and for magnetic impurity with ordinary tip. 
Such suppression of LDOS is a direct consequence of the 
absence of backscattering of helical waves due to TRS. 
Correspondingly in the fourier transform of LDOS, there 
is a sharp peak at k = 2kp for power law, which is 
absent for R~^ power law, as shown in Fig|31 For mag- 
netic impurities with spin-polarized tip, the first term in 
Eq. dominates no matter whether the pair of station- 
ary points is TRS or not (due to the contribution of the 
s = a term) , which gives the visibility of the TRS scat- 
tering wave vector qi . This distinct response of surface 
states to magnetic impurities from that of nonmagnetic 
impurities provides a crucial criteria for the breaking of 
TRS on the surface of TlSf^i Secondly, in the discussion 
above we have assumed the matrix element SJ^^ to be 
nonzero if it is not forbidden by time-reversal symmetry. 
There may be some other reasons for the matrix clement 
to vanish. For example, the states at two TRS stationary 



points have opposite spin. If the impurity spin happens 
to be parallel (or anti-parallel) to their spin, the matrix 
element S^„j can vanish. For non-TRS stationary points, 
this may occur accidentally, but generically the spin of 
the two states n and m are not parallel, so that the ma- 
trix clement is nonvanished for any impurity spin. Since 
such zeros of matrix elements arc at most only realized 
for some particular directions of the impurity spin, in the 
following we will focus on the generic cases with nonzero 
matrix element as long as it is not forbidden by time- 
reversal symmetry. Thirdly, in the integral over energy, 
we have assumed Vyi , v'yj ^ so that the only poles in 
the complex energy plane are e = e' — lj + i6. However, 
in general, it is possible that there are other poles from 
Vyi = or v'yj = 0; which means the stationary points 
in CEC arc also saddle points in the energy-momentum 
dispersion. In that case, we shall further expand Vyi (or 

v'yj) around w as Vyi{e) = Vyt{uj) + {dvyi/ de){s - lj) -\ , 

and keep the first nonzero term. This won't modify the 
power laws in spatial dependencci^ Finally, note that 
when summation over the stationary-point pairs, (ij), 
we always choose the pair such that one point has posi- 
tive velocity Vyi and the other has negative velocity v'y^ 
As a summary of the discussion above, the power laws of 
LDOS for point impurity are concluded in Table. HI 

TABLE I: Power laws of Friedel oscillations for point impurity 





charge 


LDOS 


spin LDOS 


nonmagnetic TRP 


R- 


-'2 




non-TRP 


R- 


-1 




magnetic TRP 


R- 


-2 




non-TRP 


R- 


-1 


R'^ 



To provide further intuition on the result (1121) . we 
consider some simple examples. The first example is 
a 2DES without spin-orbit coupling described by the 
familiar Hamiltonian Hq = h'^k^/2m, which has two 
degenerate and isotropic Fermi surfaces, as shown in 
FigEJa). According to our theory, the main contribu- 
tion to the LDOS in this example comes from the in- 
traband scattering of the same spin orientation between 
two extremal points, which wc denote as '1' and '2'. At 
these points we have ky2 = Px2 = kg, k'y^ = p'^^ = —fee', 
k^ = (2m£/?i^)^/^, Vy2 = hky2/m, Vy^ = hk'yi/m, and 
= C22 = 1- Inserting these quantities into Eq. (fT2)) 
and keeping only to the first order expansion of T-matrix, 
we get 6pq]I {lo, Ry) ~ —(Vm'^/Tr'^h^q) cos(2gi?)/ii;, which 
has R~^ power law. Note that the interband contribu- 
tion to the LDOS in this example is from a pair of TRS 
extremal points, which has a R^^ power law. In contrast, 
in the example of a 2D Dirac CEC, Hd = 72 • (cr x k), 
there is only one non-degenerate band at a given energy 
due to the spin splitting, as shown in FiglS^b). Thus 
only intraband scattering between a pair of extremal 
TRP contributes to the LDOS, and = 0. In- 
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sorting the quantities ky2 = Px2 = e/7, Ki = P'xi = 
-e/7, and Wj,i(2) = 7Sgn[A;yi(2)]/?i into Eq.|n]), we get 
5p\^^{uj,Ry) ~ (l//47r272)sin(2gi?)/i?2, which is consis- 
tent with our expectation. 

In a recent STM measurement of the TI Bi2Te3 doped 
with Agfii clear standing waves and scattering wave vec- 
tors are imaged through FT-STS when the Fermi surface 
is of hexagram shape. It is observed that the high in- 
tensity regions are always along the f-M direction, but 
the intensity in T-K direction vanishes. This observa- 
tion can be well-understood using our stationary phase 
approximation theory. Among the wave vectors qi, q2, 
q2 and qs shown in FiglJl^b), qi and qj correspond to 
scattering between stationary points, while qa and q2 do 
not. This explains why no standing waves corresponding 
to q3 are observed in FT-STS. Within the other two, sta- 
tionary points connected by qi are also TRP which shall 
contribute the power law of according to our result. 
Therefore its intensity in FT-STS is too weak to be ob- 
served. For wave vectors q2 and qj along f-M direction, 
q2 is stationary but non-TRS. Our result shows that this 
wave vector contributes an R^^ power law, which is re- 
sponsible for the high intensity reported in Ref J^. 

B. Edge impurity 

Beside point impurities, one-dimensional line defect in 
the form of step edge has also been observed on the sur- 
face of 3DTI J^iil Magnetic edge defects can possibly be 
realized by depositing a magnetic layer on top of a 3DTI. 
In this section, we discuss the interference patterns of 
electronic waves induced by magnetic and nonmagnetic 
edge defects. 

We consider an edge defect along the x-dircction on top 
of a 3DTI surface with the Hamiltonian V{y) = VS{y)a^. 
A magnetic edge defect has been illustrated in FiglTJ The 
main difference between an edge defect and a point defect 
is the momentum conservation along the edge impurity 
orientation, which means one of the loop integrations in 
Eq. ^ should be removed. Following similar calculations 
as performed in the case of a point impurity, the LDOS 
for the edge impurity is given by 

Sp,A^,R)= - ilm / f^S,^,,,y(^-^'y^ 



X tr [G^iio, k)T''(w, fc,)G5(w, k')aH3) 



where Tf'iuj^kx) = Va^'/{l - Va^'Gl{uJ,kx)) with 
G^{uj,kx) = /^Go(w, k). Similarly as the case of 
a magnetic point impurity, the T-matrix for a mag- 
netic edge impurity can again be separated into a spin- 
dependent and a spin-independent terms. However, in 
the following discussion, we shall keep only to the first 
order expansion of the T-matrix, Va'^, which is spin- 
dependent. This simplification is appropriate for weak 
impurity potential, and it won't affect the qualitative 
conclusion of the Friedel oscillation power laws, as we 
have learned from the case of point impurities. 

In the presence of edge impurity, we arc usually in- 
terested in the LDOS in the direction perpendicular to 
the edge orientation. Similarly as the case of point im- 
purity, the LDOS in ea. p^ is first transformed into the 
diagonal basis of the topological surface bands, and then 
converted into integrations over normal and tangential 
components as in Eq.(|9l). By using the stationary phase 
approximation, now the main contribution to the loop 
integrals comes from such stationary points where their 
momentum transfer q is normal to the edge orientation, 
and the "slopes" of CEC at the two stationary points are 
the same: 



— [ky{e,kx)~k'y{e',kx)]^0. (14) 



Compared with the stationary-point condition for point 
impurity, the condition for edge impurity allows more 
possibilities. One such example is shown schematically 
as qi in FigUl^c) where the pair of stationary points 
has the same nonvanished slope. Such a pair of scat- 
tering end points is not considered as stationary points 
in the case of point impurities, but are stationary for 
edge impurities. Following the same logic as the dis- 
cussion of point impurity in the last section, the CEC 
is then expanded around the stationary points as ky = 
kyi + ai{kx - kxi) - {kx - fc„)2/2p„, and the LDOS is 
approximated by 



V 



dSr. 



de' 



{2Tr)^ oj + iS - Sn n\vy^\ J (27r)2 w + i,5 - £;„ h\v' A 



dxe '^p^ 



+ {ax + a'x'f 



(15) 



Although Ea. ((TS|) looks similar to Eg. (ITT]) in point im- 



purity case, the definition of stationary points for edge 



7 



impurity in Ea.(|14p is quite different from that of point 
impurity. Therefore, a lot more terms should be included 
in the summation of stationary-point pairs (ij) here com- 
pared with the point impurity case. By integrating out 
x{x') and energy variables, we get 

i(kyi-k'y^)B, fji^* 

+ e-'*-(a + a')'P^j/i?]^^, (16) 

where Pij = PxiP'^jl [p'x] - P") and $y -f sgn(Pjj). 
In the equation above we have assumed Vyi,Vyj ^ and 
Pxi 7^ p'xj- In other words, this result is not applicable to 
the case where the CEC near the pair of stationary points 
is nested to the second order expansion. If such nesting 
happens, the quadratic terms in the expansion of CEC 
near the stationary points cancel out exactly, and higher 
orders expansion should be employed. The power laws 
of Friedel oscillations for edge impurity are summarized 
in Table. HIl which shall be used to explain the STM 
measurements about edge impuriticsJ^iii 

TABLE II: Power laws of Friedel oscillations for edge impurity 





ordinary 


spin polarized 


nonmagnetic TRP 


7?- 


-3/2 






non-TRP 


R- 


-1/2 






magnetic TRP 


R- 


-3/2 


R- 


1/2 


non-TRP 


R- 


-1/2 


R- 


1/2 



To have a feeling of how Ea. ([T5)) works explicitly, 
again we apply it to the examples of 2DEG Hamiltonian 
Hq and 2D Dirac Hamiltonian Hjj discussed previously. 
A few lines of calculations yield that for 2D quadratic 
dispersion, S p'^^Q (uj , Ry) = (Fm^/27r^?i''g'''/^) sin(2q_R — 
^)/\/t:R, which is consistent with the experimen- 
tal observation in 2DEGi^ For 2D Dirac fermion, 
Spi]^{uj,Ry) = {V/8Tr^-f^y/¥q)sm{2qR+l)/R^/^, which 
is a consequence of the absence of backscattering in he- 
lical liquid. Information in reciprocal space can be ex- 
tracted via FT-STS similarly to the point-impurity case 
exhibited in FigUl where a notable sharp peak is present 
at fc = 2fci? for a 2DEG, but is absent for the helical 
liquid. 

In an experiment by Gomes et ai, a nonmagnetic step 
is imaged by STM topography in Sb (111) surfaccJi 
The Fermi surface consists of one electron pocket at f 
surrounded by six hole pockets in T-M direction, where 
the surface dispersion has a Rashba spin splitting. The 
measured LDOS in T-M direction is fitted by a single 
q-parametcr using the zeroth-order of Bessel function of 
the first kind, see Fig. 2(c) in RefJ^, which agrees exactly 
with our result in Table. [TTl Along T-M direction, the 



(a) -t^ (b) _^ {c) 




q = 2frp q = 2fer q = k^^ + kp_ 



FIG. 5: (Color online) Schematic CEC of (a) quadratic, (b) 
Dirac, and (c) Rashba dispersions. The spin orientations for 
each degenerate band are indicated respectively by the green 
(solid) and purple (dotted) arrows. The stationary points are 
represented by red and blue dots, which are connected by the 
scattering vector q shown as dashed arrows. The intraband 
scattering occurs between the stationary points with the same 
color, while the interband scattering occurs between those 
with different colors. 



surface band can be modeled by a Rashba Hamiltonian 
where the LDOS is dominated by interband scattering 
between a pair of non-TRS stationary points, as shown 
in Figinijc). According to our analysis, the Friedel os- 
cillation has R~^/^ power law, which is the asymptotic 
expansion of Jo{qR) at large distances. Another STM 
experiment studying the edge impurity by Alpichshev et. 
al^ is in Bi2Te3 where hexagonal warping effect exists, 
and a nonmagnetic step defect is observed on crystal sur- 
face. A strongly damped oscillation is reported when the 
bias voltage is at the energy with a convex Fermi surface 
as shown in FiglJJa) . Though no fitting of the experimen- 
tal data is estimated in this region, our results predict a 
/j;-3/2 po-y^rer law. Pronounced oscillations at higher bias 
voltages where the hexagon warping effect emerges are 
observed with R^^ fitting. Despite of the quantitative 
difference with our result of this R~^ oscillation 

has been explained in several other works22i2i beyond our 
simple model. 

The results summarized in Tables. HI and IIIl provide a 
quantitative description of the QPI by magnetic impu- 
rities in general, which include the interference between 
two orthogonal helical waves discussed in Sect. HIl as a 
particular case. The interference of helical waves cor- 
responds to the scattering between two TRS stationary 
points, like the qi's in Figs, ^a), (b) and (c). The inter- 
esting thing is that the LDOS in charge and spin chan- 
nels from the very same pair of TRS stationary points 
have quite distinct behavior. With magnetic impurities, 
the power laws of charge LDOS arc 
point and edge impurities respectively. As a result of 
TRS, the charge LDOS has higher power indices than 
the R~^ and modulations of the corresponding 

spin-polarized LDOS, which manifests the TRS break- 
ing. To distinguish the response of topological surface 
states to magnetic impurities from that of the nonmag- 
netic impuriticSfii spin-resolved STM experiments are es- 
sential. 
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C. Friedel oscillations for CEC with generic shape 

In this section, we generalize the results obtained above 
and obtain the most general formulation of the QPI on 
the surface of a 3DTI. 

In the discussion of point impurity in Scct llll we 
have focused on the case of extremal points, around 
which the expansion of the CEC has nonvanishing sec- 
ond derivatives. However it is in general also possible 
that the principle radii of the curvature of the CEC at 
the stationary points, pxij diverges so that the third or 
even higher order expansions of the CEC at the station- 
ary points should be employed. For example, when the 
stationary points are also turning points on the CEC, 
see q2 in FiglJl^b), the expansion of the CEC should be 
kept to the third order. In the case of edge impurity pre- 
sented in Sect llll Bl it is possible that Pxi,p'xj ¥^ 0, but 
Pxi = p'xj so that Pij diverges. This happens when the 
CEC near the stationary points is highly nested, and we 
need to go beyond the quadratic expansion of the CEC 
till the first power at which the two segments of the CEC 
arc not nested. 

To understand the LDOS behavior in ordinary and 
spin-resolved STM experiments in these most general sit- 
uations, we assume in general that the first nonvanishing 
coefficients in the expansion of the CEC around the sta- 
tionary points have the order I and h respectively, where 
l,h £ ISl are generically different. Then ky{£,kx) and 
ky{e',k'^) on the CEC arc expanded around the station- 
ary points separately as ky = kyi + /3f\kx — k^if and 
k'y = k'yj + Pf''\k'^ - k'^jf, where the /3's are the first 



nonzero expansion coefficients with /3f ^ {d'ky/dk'^^)/ll 



and similarly for (3] 



,(h) 



Notice that in the case of edge im- 



purity, ill = h, one more constraint P^j"' ^ should be 
further imposed on the expansion to obtain a meaningful 
LDOS. Having analyzed the properties of the station- 
ary points on the CEC, the same calculation procedures 
as performed in Sects lIIl"S1 and ITlI Bl for point and edge 
impurities can be carried out in a straightforward way, 
which leads to the following most general results for point 
impurity 



p«(w,R) cx ^^Im^^ 



Aky^-k'^R 



X] I 

t'2 



ni,mj ^ni.mj 



and for edge impurity 



V 



R' 



J.{kyi-k'y-)R 



mn ij 



\yy^VyJ\ 



{a+ar{R\f^^:i-C^\y 



Kl,l,h) 



(18) 



These two equations complete the key results in this 
work. In the above, we have used the notation min(Z, h) 
and max(^,ft,) to represent taking the minimum or the 
maximum one between I and h. The corresponding power 
laws of the Friedel oscillations in these most general cases 
are summarized in Tables. IIIII and IIVI We see that by 
taking I = h ^ 2, these results recover those exhibited in 
Tables. HI and HIl obtained in the last two sections. 



TABLE III: General power laws of Friedel oscillations for 
point impurity 

ordinary spin-polarized 
nonmagnetic TRP T^-^t + tt)" „i„(!,h) 
non-TRP R-^T + i) 

magnetic TRP R-'^T + i)- „,inl.h) R-'-T + i^ 

non-TRP R-'-r + i'' R-'-T + i^ 



TABLE IV: General power laws of Friedel oscillations for edge 
impurity 





ord 


inary 


spin-polarized 






a 






nonmagnetic TRP 


R " 


1 






non-TRP 


R " 


,ax(l,h) 
3 




1 


magnetic TRP 


R " 


,ax(l,h) 
1 


R 


iax(I,h) 
1 


non-TRP 


R " 


,ax(l,h) 


R - 


iax(I,h) 



IV. CONCLUSIONS 

In conclusion, long-distance asymptotic behavior of the 
LDOS for nonmagnetic and magnetic, point and edge im- 
purities on a generic shape CEC are derived in Eas. ([T^ . 

pT|) . and (dHl) using the stationary phase approx- 
imation approach. The corresponding power laws of 
Friedel oscillations are summarized in Tables. U to IIVI 
^ The QPI induced by surface magnetic impurities is stud- 
^ied in particular, to illustrate the fact that the inter- 
ference patterns of charge intensities are indistinguish- 
able from those of nonmagnetic impurities, while the spin 
LDOS show distinct behavior from those of nonmagnetic 
impurities. We propose a closed "magnetic wall" geome- 
try which manifests such a unique interference property 
of helical liquids. These results depend only on the TRS 
as well as the local geometry around the stationary points 
on the CEC, which provide a systematic tool for the anal- 
ysis of STM experiments for generic surface states. 
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